Dendrites as a biologically inspired approach to overcome catastrophic forgetting

Jeremy Forest

2023-10-19

Point-neurons

  • Computes a linear weighted sum of its inputs, followed by a non-linearity
  • While historically biologically-inspired, they are very far from a biological neuron

  • Point neurons are abstraction built to answer specific questions at a specific time

  • From a ‘fit to neuroscience data’ point-of-vue, it’s a bad model BUT

  • The point neuron has found its usefullness since it has become the default unit model in ANN

  • Most of the complexity is abstracted away for better computational efficiency.

  • But this neuron abstraction coupled with the way we train deep nets has shown cases where they fail miserably

    → continual learning and the catastrophic forgetting problem

  • Which details matter and which don’t ?

Relevance of complexity (1)

  • Multiple studies by many teams has point toward the importance of complexity both at the level of single neuron and in neural networks.

  • Examples (non-exhaustive):

    • Payeur et al. (2021) : Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits

      when paired with short-term synaptic dynamics, regenerative activity in the apical dendrites, and synaptic plasticity in feedback pathways, a burst-dependent learning rule can solve challenging tasks that require deep network architectures. […]well-known properties of dendrites, synapses, and synaptic plasticity are sufficient to enable sophisticated learning in hierarchical circuits.

    • Perez-Nieves et al. (2021): Neural heterogeneity promotes robust learning

      the performance of spiking neural networks trained to carry out tasks of real-world difficulty, with varying degrees of heterogeneity, and found that it substantially improved task performance.

Relevance of complexity (2)

  • Zeldenrust, Gutkin, and Denéve (2021): Efficient and robust coding in heterogeneous recurrent networks

    networks consisting of a heterogeneous population of different neuron types are both more efficient and more robust against correlated noise.

  • Jones and Kording (2021): Might a Single Neuron Solve Interesting Machine Learning Problems Through Successive Computations on Its Dendritic Tree?

    simple model where the dendrite is implemented as a sequence of thresholded linear units. We manipulate the architecture of this model to investigate the impacts of binary branching constraints and repetition of synaptic inputs on neural computation. […] models with such manipulations can perform well on machine learning tasks, such as Fashion MNIST or Extended MNIST. We find that model performance on these tasks is limited by binary tree branching and dendritic asymmetry and is improved by the repetition of synaptic inputs to different dendritic branches.

What are dendrites ?

  • Dendrites are major elements of biological neurons

  • They are projections from the cell’s soma into the surrounding environment

  • They make up most of the substrate of connections between cells (there can also be sub-structures called spines or not)

  • Dendrites have extremely diverse morphologies

(Donohue and Ascoli 2008)

  • In biology structure and function are typically linked => there should be a dendritic function explaning this diversity

Conceptual usefullness of dendrites

  • Beniaguev, Segev, and London (2021) showed that a layer 5 pyramidal neuron I/O could be approximated using a 7 layers ANN composed of point neurons.

(Beniaguev, Segev, and London 2021)

  • However, no matter how deep an ANN is, it will always be subject to catastrophic forgetting.

  • Hypothesis : Dendritic units can help minimize the catastrophic forgetting problem in continual learning while still maintaining a low compute overhead

Dendrite properties (1) : coincidence detectors and information routing

  • Dendrite can act as coincidence detectors.

  • Differing information content is routed to different part of the dendritic arbor

  • Leading to the potential emergence of functional subdomains

Dendrite properties (2) : Linear and non-linear computations

  • Neurons contains active or passive dendrites.

  • Passive dendrites show only linear or sublinear input summation.

  • Active dendrite can show supralinear outpout.

  • Same dendritic branch stimulation can results in linear or supra-linear output depending on stimulation conditions.

Dendrite properties (2) : Linear and non-linear computations

  • Same dendrite stimulation can result in non-linearity whereas that is not the case for between dendrites stimulations

(London and Häusser 2005)

Dendrite properties (3) : Dendritic plateau potentials

  • Dendrites can integrate information over long time periods

(Antic et al. 2010)

Dendrite properties (4) : Dendrite hetereogeneity

  • Dendrites functional properties depends on their morphology hence lots of room for variability / hetereogeneity in dendrites properties.

(Tran-Van-Minh et al. 2015)

Dendrites relevance to neuronal computations

Case study 1: Tzilivaki, Kastellakis, and Poirazi (2019) - Challenging the point neuron dogma: FS basket cells as 2-stage nonlinear integrators

  • Non linear dendrites in interneurons help store memories using fewer ressources
  • Linked memory are stored in common dendrites, through synapse clustering

Dendrites relevance to neuronal computations

Case study 2: Jones and Kording (2021) - Might a Single Neuron Solve Interesting Machine Learning Problems Through Successive Computations on Its Dendritic Tree?

  • Modelling dendrites as a sequence of thresholded linear unit solves typical ML benchmarks (non-CL scenarios)

  • Benefit from having the same input onto several branches of the dendritic tree = importance of synaptic input repetition

  • Performances on-par with a fully-connected neural network (FCNN)

Dendrites relevance to neuronal computations

Case study 3: Grewal et al. (2021) - Going Beyond the Point Neuron: Active Dendrites and Sparse Representations for Continual Learning and Iyer et al. (2022) - Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments

  • Adding dendrite abstractions in an MLP help preventing catastrophic forgetting both in abstract task (permutedMNIST) and reinforcement learning (RL) tasks.

  • Context-driven task-specific subnetworks activation gated by dendritic inputs

Dendrites relevance to neuronal computations

Case study 3: Grewal et al. (2021) - Going Beyond the Point Neuron: Active Dendrites and Sparse Representations for Continual Learning and Iyer et al. 2022 - Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments

  • Dendrite specialize in few tasks.

  • Contradict the hypothesis that a dendritic neuron is equivalent to a multilayer network as classical deep networks are incapable of performing well due to catastrophic forgetting, regardless of network depth

Conclusion

  • Dendrites’ computational role is far from solved

  • Neuroscience informs models design which in turn inform back neuroscience. That’s a neat loop

  • Depending upon the task, adding dendrites could increase computational overhead without benefits i.e. in non continual learning scenario there might be few, if any, benefits to adding dendrites

  • In continual learning settings, dendrites could be a powerfull mechanism to design networks more robust to catastrophic forgetting

  • There is still a lot to open ended questions here.

    • Is the computational overhead of dendrite worth the cost ?
    • Which are the scenarios where having dendrites is helping most to perform the task ?
    • What are the dendritic mechanisms that would enhance classical deep nets the most ?

Thank you — Questions ?

References

Antic, Srdjan D., Wen-Liang Zhou, Anna R. Moore, Shaina M. Short, and Katerina D. Ikonomu. 2010. “The Decade of the Dendritic NMDA Spike.” Journal of Neuroscience Research 88: 2991–3001. https://doi.org/10.1002/jnr.22444.
Ariav, Gal, Alon Polsky, and Jackie Schiller. 2003. “Submillisecond Precision of the Input-Output Transformation Function Mediated by Fast Sodium Dendritic Spikes in Basal Dendrites of CA1 Pyramidal Neurons.” Journal of Neuroscience 23 (21): 7750–58. https://doi.org/10.1523/JNEUROSCI.23-21-07750.2003.
Beniaguev, David, Idan Segev, and Michael London. 2021. “Single Cortical Neurons as Deep Artificial Neural Networks.” Neuron 109 (17): 2727–2739.e3. https://doi.org/10.1016/j.neuron.2021.07.002.
Deng, Lei, Yujie Wu, Xing Hu, Ling Liang, Yufei Ding, Guoqi Li, Guangshe Zhao, Peng Li, and Yuan Xie. 2020. “Rethinking the Performance Comparison Between SNNS and ANNS.” Neural Networks 121 (January): 294–307. https://doi.org/10.1016/j.neunet.2019.09.005.
Donohue, Duncan E., and Giorgio A. Ascoli. 2008. “A Comparative Computer Simulation of Dendritic Morphology.” PLOS Computational Biology 4 (6): e1000089. https://doi.org/10.1371/journal.pcbi.1000089.
Grewal, Karan, Jeremy Forest, Ben Cohen, and Subutai Ahmad. 2021. “Going Beyond the Point Neuron: Active Dendrites and Sparse Representations for Continual Learning.” https://doi.org/10.1101/2021.10.25.465651.
Iyer, Abhiram, Karan Grewal, Akash Velu, Lucas Oliveira Souza, Jeremy Forest, and Subutai Ahmad. 2022. “Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments.” Frontiers in Neurorobotics 16 (April): 846219. https://doi.org/10.3389/fnbot.2022.846219.
Jones, Ilenna Simone, and Konrad Paul Kording. 2021. “Might a Single Neuron Solve Interesting Machine Learning Problems Through Successive Computations on Its Dendritic Tree?” Neural Computation 33 (6): 1554–71. https://doi.org/10.1162/neco_a_01390.
London, Michael, and Michael Häusser. 2005. DENDRITIC COMPUTATION,” 32.
Major, Guy, Matthew E. Larkum, and Jackie Schiller. 2013. “Active Properties of Neocortical Pyramidal Neuron Dendrites.” Annual Review of Neuroscience 36 (July): 1–24. https://doi.org/10.1146/annurev-neuro-062111-150343.
Payeur, Alexandre, Jordan Guerguiev, Friedemann Zenke, Blake A. Richards, and Richard Naud. 2021. “Burst-Dependent Synaptic Plasticity Can Coordinate Learning in Hierarchical Circuits.” Nature Neuroscience 24 (7): 1010–19. https://doi.org/10.1038/s41593-021-00857-x.
Perez-Nieves, Nicolas, Vincent C. H. Leung, Pier Luigi Dragotti, and Dan F. M. Goodman. 2021. “Neural Heterogeneity Promotes Robust Learning.” Nature Communications 12 (1): 5791. https://doi.org/10.1038/s41467-021-26022-3.
Sehgal, Megha, Daniel Almeida Filho, George Kastellakis, Sungsoo Kim, Jinsu Lee, Sunaina Martin, Irene Davila Mejia, et al. 2021. “Co-Allocation to Overlapping Dendritic Branches in the Retrosplenial Cortex Integrates Memories Across Time.” bioRxiv. https://doi.org/10.1101/2021.10.28.466343.
Spruston, Nelson. 2008. “Pyramidal Neurons: Dendritic Structure and Synaptic Integration.” Nature Reviews. Neuroscience 9: 206–21. https://doi.org/10.1038/nrn2286.
Tran-Van-Minh, Alexandra, Romain D. Cazé, Therése Abrahamsson, Laurence Cathala, Boris S. Gutkin, and David A. DiGregorio. 2015. “Contribution of Sublinear and Supralinear Dendritic Integration to Neuronal Computations.” Frontiers in Cellular Neuroscience 9. https://www.frontiersin.org/articles/10.3389/fncel.2015.00067.
Tzilivaki, Alexandra, George Kastellakis, and Panayiota Poirazi. 2019. “Challenging the Point Neuron Dogma: FS Basket Cells as 2-Stage Nonlinear Integrators.” Nature Communications 10 (1): 3664. https://doi.org/10.1038/s41467-019-11537-7.
Zeldenrust, Fleur, Boris Gutkin, and Sophie Denéve. 2021. “Efficient and Robust Coding in Heterogeneous Recurrent Networks.” PLoS Computational Biology 17 (4): e1008673. https://doi.org/10.1371/journal.pcbi.1008673.